Hand hygiene device, system, and method

ABSTRACT

In one embodiment, a device obtains sensor data captured by one or more sensors deployed in a washroom, the sensor data comprising audio signals. The device identifies, using the sensor data as input to one or more machine learning-based classifiers, one or more events associated with a user of the washroom. The device makes a decision as whether the one or more events comply with a hand hygiene protocol. The device provides data indicative of the decision for presentation to the user.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 63/003,514 filed Apr. 1, 2020, the entire contents ofwhich is hereby incorporated by reference as if fully set forth herein.

TECHNICAL FIELD

The present disclosure generally relates to sensor systems and sensoranalytics. More specifically, embodiments of the disclosure relate todevices, systems, and methods for hand hygiene.

BACKGROUND

Regular hand washing is considered the most important precaution toreduce the chances of contamination from germs and viruses, such as thecoronavirus known as COVID-19, lingering on surfaces. In the era ofreducing the spread of germs and viruses, there is a heightened interestin improving compliance with this form of personal hygiene—sometimesreferred to as hand hygiene. Indeed, the U.S. Centers for DiseaseControl and Prevention (CDC) states handwashing is one of the mosteffective ways to cut the spread of infectious diseases. Dirty hands areestimated to contribute to 50 percent of all foodborne illnessoutbreaks.

Most handwashing facilities today are actually quite antiquated, withmany washrooms simply including a sink, soap, and some form of mechanismfor drying one's hands (e.g., towels, air dryers, etc.). However, newopportunities now exist to modernize such facilities, thanks to therecent proliferation of sensor technologies. In addition to introducingnew sensor systems for handwashing facilities, custom data analytictechniques are also introduced herein that work in conjunction withthese sensor systems, to improve today's handwashing facilities and cangreatly help to reduce the spread of contagious diseases.

BRIEF DESCRIPTION OF DRAWINGS

The example embodiment(s) of the present disclosure are illustrated byway of example, and not in way by limitation, in the figures of theaccompanying drawings in which like reference numerals refer to similarelements and in which:

FIGS. 1a and 1b illustrate, respectively, a frontal and side view of anexemplary embodiment of a device embodying the present disclosure.

FIG. 1c illustrates an example location of the device relative to awashroom sink.

FIG. 2 illustrates a wearable embodiment of the device.

FIG. 3a illustrates a block diagram of an embodiment of the device.

FIG. 3b illustrates a block diagram of the monitoring of an ambientenvironment.

FIG. 4a contains a table illustrative of exemplary of trigger/eventsounds and motions in the ambient environment of a washroom.

FIG. 4b contains a table illustrative of exemplary of the device outputreinforcement according to one or more embodiments.

FIG. 4c contains a table illustrative of the voice messages the devicemay detect and recognize according to one or more embodiments.

FIG. 5a illustrates an exemplary flow chart of the device in operationdetermining whether user activity is related to hand hygiene.

FIG. 5b illustrates an exemplary flow chart of the processes of a firstembodiment of the device in operation.

FIGS. 6a and 6b illustrate an exemplary flow chart of the processes of asecond embodiment of the device in operation.

FIGS. 7a-7m illustrate exemplary display outputs of the device.

FIG. 8 illustrates a flow chart of the configuration processes of thedevice.

FIGS. 9a-9c illustrate exemplary user interfaces for a companion mobileapp.

FIGS. 10a-10e illustrate exemplary audio profiles of various washroomelements.

While each of the figures illustrates a particular embodiment forpurposes of illustrating a clear example, other embodiments may omit,add to reorder, and/or modify any of the elements show in the figures.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the example embodiment(s) of the present disclosure. Itwill be apparent, however, that the example embodiment(s) may bepracticed without these specific details. In other instances, well-knownstructures and devices are shown in block diagram form in order to avoidunnecessarily obscuring the example embodiment(s).

As would be appreciated, sensor systems are introduced herein that canbe deployed to handwashing facilities, to capture various sensor data.According to various embodiments, as detailed further below, machinelearning, statistical, and/or heuristic techniques can be used on thecaptured sensor data for purposes of ensuring compliance withrecommended handwashing guidelines, reporting, and/or other functions.

In general, machine learning is concerned with the design and thedevelopment of techniques that recognize complex patterns in a set ofinput data, such as sensor data captured by the sensor systems herein.In various embodiments, example machine learning techniques that can beused to implement the techniques herein may include, but are not limitedto, nearest neighbor (NN) techniques (e.g., k-NN models, replicator NNmodels, etc.), statistical techniques (e.g., Bayesian networks, etc.),clustering techniques (e.g., k-means, mean-shift, etc.), neural networks(e.g., reservoir networks, artificial neural networks, etc.), supportvector machines (SVMs), logistic or other regression, Markov models orchains, principal component analysis (PCA) (e.g., for linear models),singular value decomposition (SVD), multi-layer perceptron (MLP)artificial neural networks (ANNs) (e.g., for non-linear models), deeplearning neural networks (DNNs), replicating reservoir networks (e.g.,for non-linear models, typically for time series), random forestclassification, convolutional networks (CNN), or the like.

In various embodiments, the data analytics techniques herein may beperformed using one or more supervised, unsupervised, or semi-supervisedmachine learning models. Generally, supervised learning entails the useof a training set of data that is used to train the model to applylabels to the input data. For example, the training data may includesample sensor that has been labeled as being indicative of compliant ornon-compliant with acceptable handwashing protocols. On the other end ofthe spectrum are unsupervised techniques that do not require a trainingset of labels. Notably, while a supervised learning model may look forpreviously seen patterns that have been labeled as such, an unsupervisedmodel may instead look to whether there are sudden changes or patternsin the behavior of the metrics. The unsupervised models may learn fromdata it takes in but not train prior to deployment. Semi-supervisedlearning models take a middle ground approach that uses a greatlyreduced set of labeled training data, allowing training data to beextrapolated from a smaller set of labeled data.

Handwashing appears to be a simple well-bounded problem; however,depending on the accuracy of ensuring a user's compliance withrecommended handwashing protocol, the problem becomes a more complex andchallenging technical problem. There are, in fact, several actions auser may take that deviate from the recommended handwashing protocol.Static signage describing the protocol is sometimes not sufficientlyunderstood, and users are not held accountable in real-time to complywhen they deviate from the recommended handwashing protocol. One or moregoals of the present disclosure are to increase hand hygiene compliance,namely: (1) to encourage a user to wash his/her hands in a particularenvironment where the device is situated, and (2) to guide users to washtheir hands in the recommended protocol.

Some handwashing actions a user may take include: start the handwash;stop the handwash; proper lathering; improper lathering; touching offaucet handles with hands; touching of a tap handles with somethingother than hands such as an elbow; touching of faucet tap handles with atowel; use of soap; handwashing with no use of soap; drying hands with atowel or hand air-dryer; not drying hands with a towel or hand airdryer, etc. However, the user's behavior is influenced by numerousfactors. Factors may include the social need to participate in the needto help the collective good to stop the spread of viruses, and diseases;the cleanliness of the washroom; the availability of soap, towels and/orair dryers; user knowledge of recommended handwashing protocol;accountability imposed on the user; some form of influence/oversightthat encourages the user to comply with a recommended handwashingprotocol, or some combination thereof.

The device, system, and method described herein describe techniques tohelp encourage compliance with a recommended handwashing protocol byinstructing the user on the protocol steps by displaying appropriateinstructions in real-time when the user is about to wash his/her hands.Depending on the embodiment of the present disclosure, variousadvantages may result. The protocol instructions may be in the form ofimages, animations, video, or some combination thereof. This outputassists certain types of learners to better understand and comply withthe recommended handwashing protocol. Advantageously, the device isprogrammed or configured to provide this protocol instructioncontemporaneously when the user is or about to wash his/her hands.

Further, functioning washrooms elements such as toilets, faucets, soapdispensers, and sinks do not necessarily require replacement ormodification. Also, the device requires limited space and provides theintended compliance without requiring physical interaction with thedevice. Further advantageously, the user need not be expected toremember the best steps to properly wash his/her hands but is remindedof these recommended steps contemporaneously when washing his/her hands.When the user deviates from the recommended handwashing protocol, thedevice may be programmed or configured to modify its behavior toencourage the user to comply with the recommended handwashing protocol.For instance, the system may evaluate compliance with the recommendedhandwashing protocol set out by the World Health Organization (WHO):

-   -   Step 1: Wet hands with safe running water    -   Step 2: Apply enough soap to cover wet hands    -   Step 3: Scrub all surfaces of the hands for at least 20 seconds    -   Step 4: Rinse thoroughly with running water    -   Step 5: Dry hands with a clean, dry cloth, single-use towel, or        hand drier as available    -   Step 6: Turn off water with a towel or elbow.

Step 3 above is broken into substeps to ensure the cleaning of allsurfaces by recommending the following scrubbing steps:

-   -   Step 3a: Rub hands palm to palm;    -   Step 3b: Rub right palm over left dorsum with interlaced fingers        and vice versa;    -   Step 3c: Rub palm to palm with fingers interlaced;    -   Step 3d: Rub backs of fingers to opposing palms with fingers        interlocked;    -   Step 3e: Rotational rubbing of left thumb clasped in right palm        and vice versa; and,    -   Step 3f: Rotational rubbing, backward and forwards with clasped        fingers of the right hand in left palm and vice versa.

Alternatively, the techniques herein may be used to enforce other handhygiene protocols, as desired.

FIGS. 1a-c

FIG. 1a is an example of a physical embodiment of device 300 in afrontal view. The display 320 presents an illustrative example of onestep of several recommended steps to properly wash hands, namely Step 2:Apply soap to the hands. FIG. 1b illustrates an example of the side viewof device 300 showing the housing 302, display 320, wall bracketmechanism 303, audio capture device (microphone) 310, an audio outputdevice (speaker) 315 according to one or more embodiments. FIG. 1cillustrates the placement of device 300 relative to the sink 150 andmirror 175 according to one or more embodiments. Device 300 may beinstalled in various locations, including on or in the mirror, on thesink counter, or affixed to an adjacent wall to the sink. A goal is toensure the user has a visual line of sight to the display 320 of thedevice when in operation.

FIG. 2

FIG. 2 shows a user wearable embodiment of the presentdisclosure—namely, a smartwatch device 200. The smartwatch device 200may work as a companion device with other devices such as a mobile phone(not shown) or device 300. The device 200 is programmed with thefunctions described herein and may use information communicated by theuser's mobile phone to cause certain output behavior. For example, withlocation-based services are shared by the mobile phone with the device200, the device 200 recognizes when the user returns to his/her homeafter an outdoor trip, thereby outputting instructions to the user towash his/her hands upon arrival home. The wearable device 200 alsodetects handwashing motion to trigger output to encourage adherence tothe recommended time for washing the user's hands. While FIG. 2 depictsa smartwatch device 200, in other embodiments, the present disclosure'stechniques can be implemented with different types of wearables instead,such as smart glasses, a fitness tracker, a smart ring, etc.

FIGS. 3a and 4a

FIG. 3a illustrates a block diagram of example device 300 in which thetechniques described herein may be practiced, according to someembodiment. Device 300 is programmed or configured to encouragehandwashing and compliance with a recommended handwashing protocol. FIG.3a illustrates only one of many possible arrangements of componentsconfigured to execute the programming or functionality described herein.Other arrangements of components, and the division of work between thecomponents may vary depending on the arrangement. The diagramsystematically illustrates a device 300 to receive movement/audio inputsensor data and to process the sensor data through analytics accordingto one or more embodiments. FIG. 3a shows a system for detecting certainsounds and/or motions by the device according to one aspect of thepresent disclosure. The system includes an electronic device 300 thatmay be executing software that is monitoring for a trigger. The termtrigger or event is used generically here as referring to any soundand/or motion input that may be detected by one or more I/O DeviceInterfaces 325 and that may cause, based on that detection, the softwareprogrammed in the device 300 to respond or take specific actions toencourage hand hygiene compliance.

With reference to FIG. 4a , a table contains examples of trigger sounds,in the first column, in the ambient environment. These sounds are thosetypically found near a washbasin/sink, such as in a washroom. The termswashroom, bathroom, toilet room, and water closet are usedinterchangeably herein and refer generally to a space including at leasta handwashing sink or basin. One skilled in the art appreciates manyother sounds may be recognized and are within the scope of the presentdisclosure. If, for example, the device 300 is set next to the entrywayof a home, the trigger sounds include entryway doors and garage doorsopening and closing. FIG. 4a also includes in the second column of thetable examples of trigger motions in the ambient environment that mayrecognized by the device 300. The listed motions do not necessarily haveany relation to the first column of sounds. Further examples of triggermotions recognized by device 300 may include the hand motionsillustrated in FIGS. 7a -7 l.

The block diagram of device 300 is not limited to the existingcomponents illustrated in FIG. 3a . For example, although not shown, thedevice will preferably include a power source, such as arechargeable/disposable battery or connection to the building's AC/DCpower system, to power the device 300. The current components may beused in the construction of the device 300. The device 300 may also notbe limited to just the quantity of components displayed in FIG. 3a . Thedevice 300 could be subject to multiple audio capture devices 310,multiple controllers/processors 330, and multiple device interfaces 325to improve device 300 performance if needed. The device 300 may also beconstructed in a way to have the components connected and/orcommunicating to the “cloud.”

Device 300 may contain a motion detector capture device 305 that willdetect motion in and around device 300. Various types of sensors may beused for this purpose, including passive motion sensors (such as PIRsensors) and active motion sensors (such as LIDAR (Light Detection andRanging) sensors).

Device 300 may contain an audio capture device 310. The audio capturedevice 310 may contain one or more microphones (e.g., an array ofmicrophones) which are transducers configured to receive a sound fieldin the ambient environment of the device 300, and in response, provideone or more audio signals corresponding to it.

Device 300 may also contain audio output devices 315 for producingnotification sounds and/or audio messages and/or instructions. Thedevice 300 may contain a video output device 320, such as e-ink display,LCD, LED and projection displays, to display images, video, and/or text.

Device 300 may include an address/data bus 365 to transfer informationthroughout all components within the device 300. Each component withinthe device 300 will be connected to other components across theaddress/data bus 365. The device 300 may include a controller/processor330, which may serve as the central processing unit for processing dataand computer-readable instructions. The memory within the device 300 mayinclude RAM, ROM, and other types of memory. The device 300 may alsocontain a data storage 340 component for storing instructions and data.The device 300 may also be connected to removable or external memorystorage (not shown) through input/output device interfaces. The wirelesscommunications circuitry 342 may be one or more integrated circuitsand/or other hardware components configured to transmit and/or receivedata with a cellular network, a short-range network (such asBluetooth)/or a WiFi network. Thus, the wireless communicationscircuitry 342 may include components (such as one or more basebandprocessors, digital signal processing (DSP) circuits, transceivers,radio frequency (RF) front ends, and/or other components) that areconfigured for communication with the appropriate network. In variousembodiments, the wireless communications circuitry 342 may includecomponents for communicating (a) only with cellular networks, (b) onlywith short-range networks such as Bluetooth, (c) only with WiFinetworks, or (d) for communicating with some combination thereof. Thewireless communications circuitry 342 may be coupled to one or moreantennas 344. Whenever it is described herein that the device 300communicates data to/from a wireless network to another network such asthe “cloud,” communication may be performed using one or more antennas.The device 300 includes input/output device interfaces 325 incommunication with various input/output devices. The device 300 and itsvarious components may be executed by the controller/processor and bestored in the memory, storage, an external device, or in memory/storageincluded in an Audio Recognition module (ARM) 345 discussed below.

The ARM 345 is a system within the device 300 used for processing audiosignals from sounds into triggering software in the controller/processor330 to activate certain responsive outputs by device 300. Outputscomprise informational displays (shown in FIGS. 7a-7m ) via the videooutput device 320 and/or audio output device 315. The ARM 345 mayinclude a sound classifier system 350 based on a machine learning (ML)model 355 configured to recognize different audio signals detected inthe ambient environment of the device 300. ARM 345 may store the audiosignals in a sound storage 360 to improve the device's 300 soundrecognition quality.

Although not shown, the system includes a plurality of featureextractors or feature extraction components or modules that may be inthe form of software stored in the memory of the device 300 and isexecuted by the processor 330. The feature extractors digitally processthe audio signals so as to determine various features (characteristics)of the audio signal. Extracted audio features may be used to classifycertain sounds to determine whether user is following the recommendedhandwashing protocol by the CDC. These features may also be described asthose of the sound or sound field picked up in the audio signalsassociated with those trigger sounds, such as those set out in the tablein FIG. 4 a.

The system's classifier 350 or sound classification module as softwarebeing executed by the processor, employs or otherwise includes a machinelearning model 355, though other statistical models and algorithms canbe used without departing from the scope of the present disclosure.Also, the system includes a database 360 that stores historical sounddata or information, such as previously stored sound metadata, that canbe accessed by the classifier 350. The classifier 350 determines aclassification of the audio signal based upon the audio signal'sdetermined features and, preferably, also based on the localized andlabeled sound samplings provided at the installation of various triggersounds in the ambient environment of the device 300's installationlocation. In one variation, the classifier 350 makes a binaryclassification and determines whether the audio signal relates to atrigger sound versus a non-trigger sound. However, the classifier couldalso classify a sound as being one of a number of other classifications(e.g., one or more of three or more classifications) orsub-classifications without departing from the scope of the presentdisclosure.

The device 300 may train on an audio signal dataset to enhance itsprecision in methods to encourage compliance with recommendedhandwashing protocol. By way of example, when the ARM 345 recognizes thesound of water flowing, ARM 345 triggers software in the processer 330to display what could be shown in FIG. 7b —namely: “Apply soap tohands.” The ARM advantageously also provides insight to device's 300hand hygiene compliance algorithm where ambient environment motioncannot be detected by the field of sight of the motion detector capturedevice 305. For example, the ARM may be programmed to recognize the useof a toilet by just the sound of the flush of the toilet cistern wellbefore any motion can be detected by the Motion Recognition Module (MRM)of a user walking past the device 300 after the toilet use—to bediscussed next.

Although not shown, an MRM is used for processing motion detected in theambient environment into triggering software in the controller/processor330 to activate certain responsive outputs by device 300, including butnot limited to the informational displays (as shown in FIGS. 7a-7m ) viathe video output device 320 and/or audio output device 315. Dataassociated with motion detected by the motion detector capture device305 is input for the MRM. The MRM may include a machine learningclassifier system configured to recognize different motions found in thewashroom. MRM may store the motion in motion storage (not shown) toimprove the device's 300 motion recognition quality. Set-up of the MLimplementation for enhancing motion recognition would include a datasetof thousands of recordings of different motions commonly found in andaround the washbasin area. As shown in FIG. 8 (to be described in laterdetail), device 300 is preferably initialized during the set-up withmotions commonly found in a particular washroom. The device trains on amotion dataset to enhance its precision to better recognize when acertain triggering motion occurs. Device 300 could be further enhancedby the correlation and/or associations of detecting specific sounds andmovements through the implementation of machine learning of recognizingparticular sounds and motion to trigger proactive output behavior by thedevice 300 to encourage hand hygiene compliance. By way of example, whenthe ARM recognizes the sound of the toilet flush and the MRM recognizesthe motion of a user is about to exit the washroom facility without theARM recognizing the sound of water at the sink before the attemptedexit, these conditions trigger software in the processer 330 to activateaudio and/or visual reminder to “Wash Your Hands”. The reminder may alsobe triggered at a second device in communication with a first device.The second device may be configured to include a limited set ofcomponentry found in the first device.

Privacy

There is a natural expectation by users of privacy and anonymity in manyprivate environments—especially a washroom environment. Naturally, theuse of any electronics in such environments may raise escalate theprivacy concern. One aspect of the present disclosure is the gatheringand use of data available from various sources including the ambientenvironment where the device 300 is situated to classify sounds andmotions, and to improve the accuracy of classifying those sounds andmotions as being hand hygiene-related activities. When not permitted todo so under applicable rules and regulations, the present disclosure, inone aspect, mitigates the risk of obtaining personally identifiableinformation (PII) of a user in the ambient environment of the device300. In another variation of the present disclosure, it contemplatesthat in some instances, this gathered data may include PII data thatuniquely identifies or can be used to contact or locate a specificperson. The present disclosure recognizes that the use of such personalinformation data, in the present technology, can be used to the benefitof users. For example, the personal information data can be used toprovide customized learning opportunities/feedback to a specific user tobetter adhere to acceptable hand hygiene protocols. For a specific user,the disclosure considers reducing expectations for a user that hashistorically was the least compliant user out of a larger known set ofusers. In that instance, the system outputs hand hygiene compliance thatwould gradually bring the particular user to full compliance over alonger period of instances where the device recognizes the user.

The present disclosure contemplates that the entities responsible forthe collection, analysis, disclosure, transfer, storage, or other use ofsuch personal information data will comply with well-established privacypolicies and/or privacy practices. In particular, such entities shouldimplement and consistently use privacy policies and practices that aregenerally recognized as meeting or exceeding industry or governmentalrequirements for maintaining personal information data private andsecure. Such policies should be easily accessible by the entity thatinstalls the present device and should be updated as the collectionand/or use of data changes. Personal information from users should becollected for legitimate and reasonable uses of the entity and notshared or sold outside of those legitimate uses. Further, suchcollection/sharing should occur after receiving the informed consent ofthe users. Additionally, such entities should consider taking any neededsteps for safeguarding and securing access to such personal informationdata and ensuring that others with access to the personal informationdata adhere to their privacy policies and procedures. Further, suchentities can subject themselves to evaluation by third parties tocertify their adherence to widely accepted privacy policies andpractices. In addition, policies and practices should be adapted for theparticular types of personal information data being collected and/oraccessed and adapted to applicable laws and standards, includingjurisdiction-specific considerations. For instance, in the US,collection of or access to certain health data may be governed byfederal and/or state laws, such as the Health Insurance Portability andAccountability Act (HIPAA); whereas health data in other countries maybe subject to other regulations and policies and should be handledaccordingly. Hence different privacy practices should be maintained fordifferent personal data types in each country.

Despite the foregoing, the present disclosure also contemplatesembodiments in which users selectively block the use of, or access to,personal information data. That is, the present disclosure contemplatesthat hardware and/or software elements can be provided to prevent orblock access to such personal information data. For example, in the caseof continuous audio collection (“always listening”) and storage ofhistorical sound data, the present technology can be configured to allowusers to select to “opt in” or “opt out” of participation in thecollection of personal information data during registration for servicesor anytime thereafter.

Moreover, in an embodiment, personal information data should be managedand handled in a way to minimize risks of unintentional or unauthorizedaccess or use. Risk can be minimized by limiting the collection of dataand deleting data once it is no longer needed. In one variation, thepresent disclosure utilizes digital obfuscation to abstract away anyactual or potential PII of users using privacy-enhancing technology(PET). PETs are a broad range of technologies (hardware and/or software)that are designed to extract data value in order to unleash its fullcommercial, scientific and social potential, without risking the privacyand security of this information. Two major types of PETs arecryptographic algorithms and data masking techniques. Cryptographicalgorithms include homomorphic encryption, secure multi-partycomputation (SMPC), differential privacy, and zero-knowledge proofs(ZKP). Data masking techniques include obfuscation, pseudonymization,data minimization and communication anonymizers. Any of theaforementioned PETs may be utilized in the present disclosure.

The ARM and MRM are programmed to filter any PII data artifacts capturedin the raw audio and motion input by utilizing a PET (see discussion inFIG. 3b regarding the PET component 410 and 430). Component choice mayhelp mitigate these risks. For example, using LIDAR sensor technology aspart of the motion detector capture device 305 can be set to remove allPII. A point cloud sensor data, generated by the LIDAR, is inherentlyanonymous to protect users' privacy while also giving the device's MRM ahigh-resolution image to work with. To add layer of privacy, the deviceuses edge computing systems to process data in the field and pass onanonymized object data to the cloud-based networks only if necessary. Inaddition, and when applicable, data de-identification can be used toprotect a user's privacy. De-identification may be facilitated, whenappropriate, by removing specific identifiers (e.g., name, etc.),controlling the amount or specificity of data stored (e.g., collectinglocation data a city level rather than at an address level), controllinghow data is stored (e.g., aggregating data across users), and/or othermethods.

Therefore, although the present disclosure broadly covers use ofpersonal information data to implement one or more various disclosedembodiments, the present disclosure also contemplates that the variousembodiments can also be implemented without the need for accessing suchpersonal information data. That is, the various embodiments of thepresent technology are not rendered inoperable due to the lack of all ora portion of such personal information data. For example, sound andmotion classification can be performed based on non-personal informationdata or a bare minimum amount of personal information.

FIG. 3b

Monitoring of Ambient Environment

With reference to FIG. 3b , a block diagram representing the logical,algorithmic components the device will execute when monitoring theambient environment for hand hygiene non-compliance. It is beunderstood; the reverse could be monitored—namely, monitoring theambient environment for hand hygiene compliance. For brevity, the formeris what is described herein. FIG. 3b shows a monitoring system receivingaudio signals 405 from at least one audio sensor, such as a microphone,310 and motion signals 420 from at least one motion detection sensor305. The monitoring system comprises an Audio Recognition Module forclassifying audio signals and a Motion Recognition Module forclassifying motion signals according to one aspect of the presentdisclosure. Although shown in FIG. 3b as a single component, the systemcan include a plurality of feature extractors 410 or one or more featureextraction or detection components or modules that perform specificsignal processing on received audio signals 405 to determine specificfeatures, aspects, or characteristics thereof. The plurality of featureextractors 410 can apply algorithms or modeling (e.g., including machinelearning modeling) to the audio signals (e.g., components thereof) todetermine their specific features. The plurality of feature extractors410 may include a sound class feature extractor, a directionalinformation feature extractor, and a distortion feature extractor (notshown). Audio feature extraction may utilize one or more models commonlyused in NLP (Natural Language Processing) or ASR (Automatic SpeechRecognition) such as Artificial Neural Network (ANN), Gaussian MixtureModelling (GMM), or Support Vector Model (SVM). The mentioned models andsimilar models may be used for audio signal processing and other featureextractions to determine key audio information such as MFCC(Mel-Frequency Cepstral Coefficients), filter banks, samplingfrequencies, amplitude, Fourier transform, and more for the purpose ofclassifying audio signals. The use of publicly available tools anddatasets, such as Librosa, may be used to conduct an in-depth audioanalysis and digital signal pre-processing.

In one variation, the sound class feature extractor can apply one ormore algorithms or models to the audio signals 405 to determine one ormore sound classes present in one or more of the audio signals 405. Thesound classes can include a specific sound type, such as speech, music,laughter, sounds made by particular objections (e.g., toilets flushingor other exemplary sounds set out in FIG. 4a ), etc., or other types ofsounds. In addition, the sound class feature extractor can determinewhether the audio signals include multiple sound classes or whether thesound classes vary or change over time.

The sound class feature extractor itself can include one or more machinelearning models or other (semi-)supervised learning or statisticalmodels that are trained or calibrated to determine the different soundclasses, respectively, based on the audio signal 405 as its input. Forexample, a data corpus, including various ground truth-labeled sounds ofdifferent sound classes, is collected. The data corpus can then bepartitioned or otherwise separated into training sets and testing sets.The machine learning model is trained or calibrated to determine thesound class or sound classes using the training set. The accuracy of themachine learning may then be determined using the testing set, e.g., todetermine whether the machine learning model assigns classes to signalsof the testing set at a threshold rate or level of accuracy.

Further, in one variation, a plurality of feature extractors 410 caninclude a directional feature extractor (not shown). The directionalfeature extractor can perform signal processing on one or more of theaudio signals 410 to determine spatial signatures or characteristics,e.g., there is a dynamic sound source vs. a static sound source recordedin the audio signals 410. For example, an audio signal that contains arecording of a sound emitted from a stationary device (e.g., toilet,sink water faucet, shower, bathtub, or ceiling fan) may have stationaryor static spatial characteristics, but an audio signal from a sound of aperson talking or yelling may have dynamic spatial characteristics dueto the person turning their head or walking around (while talking). Thedirectional feature extractor can process the audio signal to determinedirectional characteristics of the sound, such as a specific direction,time of arrival, angle, etc. For example, sounds emitted from suchstationary devices will generally be received from the same position andwill have the same or similar directional characteristics reflected inthe audio signal. Still further, the plurality of feature extractors 410can include a distortional feature extractor and other types of featureextractors (while still remaining within the scope of the presentdisclosure).

The ARM may include a classifier 350, employing a neural network orother suitable machine learning model 355, whose output may be a soundclassification that classifies the sound as an audio trigger vs. audionon-trigger. The classifier 350 receives the plurality of features andpreviously stored sound metadata in sound storage 360. For example,these features and historical data can be used as inputs to the neuralnetwork, deciding whether the sound is a recognized trigger audio event.In some cases, the decision made by the classifier 350 (to determinewhether the audio signal is from a trigger sound vs. non-trigger sound)can be based on just the historical data without relying upon an outputof the neural network (e.g., if the current audio signal hassubstantially similar features or the same feature vector as those of apreviously classified audio signal.)

As shown in FIG. 3b , the system may include a similar arrangement ofcomponents arranged for classifying motion/movement signals. Motionsignals generally relate to the physical movement of one or more userspresent in the ambient environment. The outcome from the motion-relatedMotion Recognition Module (MRM) has a similar goal—to classify what typeof motions detected in the ambient environment is related to handhygiene vs. those motions unrelated to hand hygiene. A plurality ofmotion feature extractors 425 or one or more feature extraction ordetection components or modules that perform specific signal processingon received motion signals 425 to determine specific features, aspects,or characteristics thereof. The plurality of motion feature extractors425 can apply algorithms or modeling (e.g., machine learning modeling)to the motion signals (e.g., components thereof) to determine theirspecific features. The plurality of motion feature extractors 425 mayinclude a motion class feature extractor and other types of motionfeature extractors (not shown). Feature extraction will require use ofML models such as PCA (Principal Component Analysis), ConvolutionalNeural Networks (CNN), SVM, K Nearest Neighbor (KNN), Random Forrest, ormodels like such. The development process of these models within the MRMmay include a workflow such as loading the data, preprocessing theimage, defining model parameters, and training/testing network. Pointcloud segmentation may take the input form of voxels, point clouds,graphs, 2D view, and other similar networks.

In one variation, the motion class feature extractor can apply one ormore algorithms or models to the motion signals 420 to determine one ormore motion classes present in one or more of the motion signals 420.The motion classes can include a specific motion type, such as userexiting washroom, a user walking toward a sink area, user washinghis/her hands, a user entering the washroom, user applying soap, userrinsing hands, a user activating sink faucet, user rinsing hands withwater, user scrubbing hands, user deactivating sink faucet, user dryinghands, etc.

The motion class feature extractor itself can include one or moremachine learning models or other supervised learning or statisticalmodels that are trained or calibrated to determine the different motionclasses, respectively, based on the motion signal 420 as its input. Forexample, a data corpus, including various ground truth labeled sounds ofdifferent motion classes, is collected. The data corpus can then bepartitioned or otherwise separated into training sets and testing sets.The machine learning model is trained or calibrated to determine themotion class or motion classes using the training set. The accuracy ofthe machine learning may then be determined using the testing set, e.g.,to determine whether the machine learning model assigns classes tosignals of the testing set at a threshold rate or level of accuracy.

The MRM may include a classifier 365 employing a neural network or othersuitable machine learning model 370, whose output may be a soundclassification that classifies the sound as an audio trigger vs. audionon-trigger. The classifier 365 receives the plurality of features andpreviously stored motion metadata in motion storage 375. For example,the neural network receives as inputs features and historical data,which can decide whether the motion is a recognized trigger motionevent. In some cases, the decision made by the classifier 365 (todetermine whether the motion signal is from a trigger motion vs.non-trigger motion) can be based on just the historical data withoutrelying upon an output of the neural network (e.g., if the currentmotion signal has substantially similar features or the same featurevector as those of a previously classified motion signal.)

Based on the sound classification decision 415 and the motionclassification decision 435, a decision is made whether the sensor datafrom the monitored ambient environment is non-compliant handhygiene-related activity. This determination may be based on a pluralityof conditions based on sound and/or motion triggers detection.Generally, if a first condition is satisfied and a second condition issatisfied, then a determination is made whether or not it isnon-compliant hand hygiene-related activity.

By way of example, let's assume the device has been configured tooperate in a washroom. If the sound classification determines that thesound processed is that of a toilet flush, and if the motionclassification determines a subsequent motion detected in the ambientenvironment is that of a user tending to exit the washroom (with afurther confirmation that the sound classifier did not detect any soundof water flowing from the sink faucet before the imminent exit), thehand hygiene-related activity decision 440 would conclude that thisactivity is that of hand hygiene-related activity particularly that ofnon-compliance to acceptable hand hygiene.

By way of another example, if the sound classification determines thatthe sound processed is that of a faucet running water into a sink, andif the motion classification and/or sound classification determines soapdispenser use was not detected, the hand hygiene-related activitydecision 440 would conclude that this activity is that of handhygiene-related activity particularly that of non-compliance toacceptable hand hygiene. The device 300 takes output action indicativeof remedying the non-compliance.

The device 300 is programmed for this sound-to-motion correlation ormapping for various permutations of trigger sounds and triggers motionsto determine whether hand hygiene activity has occurred. Naturally, amultitude of conditions are possible given the exemplary sound andmotion triggers described herein. It is naturally within the scope ofthe present disclosure that the determination could be either that handhygiene is complied with, or that hand hygiene has not to be compliedwith. In ambient environments where multiple users are present at anygiven time, a temporal user identifiable data distinguishes each userpresent to ensure that the device reacts (or outputs) according to thatparticular user's hand hygiene compliance.

FIG. 3b shows illustrate the PET components 412 and 430, which removesone or more data artifacts that may be personally identifiableinformation (PII) of a user. Although shown as a single component, thepresent disclosure may include instances of the PET 412 componentrepeated in different stages of processing the user's data information.This equally applies to the components associated with the motionrecognition module.

FIGS. 5a and 4b

FIG. 5a illustrates a flow diagram of an example process for addressinga non-compliance to a handwashing-related activity using the system setout in FIGS. 3a and 3b , according to one embodiment. FIG. 5a isintended to disclose algorithms or functional descriptions that may beused as a basis of writing computer programs to implement the functionsthat are described herein, and which cause a computer to operate in thenew manner that is disclosed herein. Further, FIG. 5a is provided tocommunicate such an algorithm at the same level of detail that isnormally used, by persons of skill in the art to which this disclosureis directed, to communicate among themselves about plans, designs,specifications and algorithms for other computer programs of a similarlevel of complexity. The steps of process set out in FIG. 5a may beperformed in any order, and is not limited to the order shown in FIG. 5a. The procedure starts with the device monitoring the ambientenvironment for user activity 450. The monitoring step may utilize theprocess and componentry set out in FIGS. 3a and 3b . A determination ismade whether or not the activity is related to handwashing or handhygiene 455. If the activity is related to hand hygiene, the processthen presents a recommended handwashing step/hand hygiene step 460.Again, the process returns to monitoring the user's activity 465, and adetermination is made if the user deviates from acceptable hand hygieneprotocol 470. If there is a deviation detected at step 470, the processproceeds to present output to the user to further encourage complianceto an acceptable hand hygiene protocol 475.

FIG. 4b lists examples of how the device modifies its output accordingto one or more embodiments. Depending on the recommended step and thetype of deviation, reinforcement may include outputting the importanceof the particular step's compliance to ensure good hand hygiene;outputting the deviation in such a manner to “call out” the deviation tothe user and to others in the washroom; repeating the display of therecommended step with additional urgency to comply with the step; orsome combination thereof. When the device 300 detects that the userappears to not comply with a recommended hand wash, the device 300highlights non-compliance to the user by the use of an audio warning, avisual warning, or some combination thereof. The audio warning could bea voice recording, “You are not quite done your handwash!” or somesuitable equivalent that may reference the type of non-compliance. Ifthe process determines the user has complied with the recommended stepat step 470, the logical procedure determines whether all hand hygienesteps are complete 480. If no, the appropriate recommended handwashingstep is outputted. If yes, the process returns to monitor the ambientenvironment for the next user 450. Before this, a variation of thepresent disclosure includes recognizing the user's hand hygienecompliance and/or feedback on improving the user's hand hygienecompliance. The device then returns to monitoring the environment todetect the next user. The steps set out in FIG. 5a may be performed inany order or simultaneously, and the steps of detecting the user and/ordetecting whether the user deviates may be continuous. Although shownsequentially, these steps may be performed in any order orsimultaneously, and at least detecting the user may be continuous.

Other remediation is available should the user fail to comply with handhygiene compliance. For example, the user ignores all recommended stepsto comply, a notification is sent to janitorial staff to clean doorhandles at that bathroom location. This especially useful a workenvironment where janitorial staff are available for this remedialeffort to mitigate the risks associated with non-compliance by users.

FIG. 5b

FIG. 5b is a flow chart illustrating an example algorithm forencouraging compliance for proper handwashing to be executed on device300, as shown in FIGS. 3a and 3b . At the start 505, the system monitorsthe environment. Monitoring may include motion sensing. Upon a userapproaching the device, which is preferably installed near the sink, thedevice will detect the user 510 and activate an output. Although shownsequentially, these steps may be performed in any order orsimultaneously, and at least detecting the user may be continuous.

Upon this activation, the device, in an embodiment, alerts the user ofthe device's 300 presence by displaying a visual message, an audiomessage/alert, or a combination thereof to the user. In anotherembodiment, the message is the statistics of those users doing a properhand wash at the sink. For example, “Over 100 people followed therecommended handwashing protocol to ensure a healthy tomorrow. You arenext!” or “Every employee has washed his/her hands today, you are next!”In this manner, the device 300 acts to influence the user's behavior tocomply with better hand hygiene through a real-time reminder to washhis/her hands.

In this embodiment, the device displays the recommended steps for aproper hand wash. Each recommended handwashing step is displayed. In analternative embodiment, the device also outputs a voice-over toreinforce each step displayed. Regarding FIG. 5, the recommendedhandwashing step of wetting the hands is displayed 515 on the device'sdisplay. Next, the recommended step of applying soap is displayed 520 onthe device's display. After that the recommended step oflathering/scrubbing the hands with soap is displayed 525 on the device'sdisplay. The message includes a countdown for hand lathering/scrubbingfor a recommended period 530 is presented to the user.

In different embodiments, the period 530 may vary depending on thecurrent public healthcare guidance. For example, in one embodiment, theperiod is twenty (20) seconds. In another embodiment, the period isfifteen (15) seconds. The configuration stage sets the recommendedperiod. The algorithm starts the timer 535, and a series of bestpractices of proper hand lathering/scrubbing is displayed 540, and thecountdown timer continues 550 until the countdown timer is complete 545.Once the recommended period of time is complete, the step of rinsing ofhands is displayed 555 on the device's display. Next, the step of dryingof hands is displayed 560 on the device's display. Next, the step ofturning off the water is displayed 565 on the device's display. Finally,the user is congratulated with a successful completion message displayed570. The process returns to start 505 by monitoring the arrival of thenext user.

FIGS. 6a & 6 b

FIG. 6a and its continuation FIG. 6b is a flow chart illustrating anexample algorithm for encouraging compliance for proper handwashing tobe executed on the device 300, as shown in FIG. 3a . At the start 605,the system monitors the washroom environment for a user 615. Upon thedevice detecting a user 610, it will activate its handwashinginformational protocol. Although shown sequentially, these steps may beperformed in any order or simultaneously, and at least detecting theuser, environment monitoring, and other deviation conditions (such asnon-compliance conditions as discussed below) may be continuous.

Upon this user detection, the device presents a message to encourageproper handwashing, namely wet the hands 615. Next, the device monitorsthe environment for running water 617. The device determines if waterfrom the faucet is detected 620. In one embodiment, the device listensthrough its audio input device 310 for the sound of running water. If nowater is detected, the device reinforces this step by re-announcing theuser to turn on the water tap to wet his/her hands 622.

Once the device detects water, the algorithm then displays instructionsto apply soap to the hands 625. Next, the device monitors theenvironment for the use of the soap dispenser 627. The device determinesif the soap is being applied to the hands 630. In one embodiment, thedevice listens for the use of the soap dispenser. If no dispenser use isdetected, the device announces to the user to use the soap dispenser632. Once soap dispenser use is detected by the device, the algorithmthen displays instructions to scrub the hands 635.

The device displays a countdown timer 637 and monitors the user'senvironment commencing the step of scrubbing 640. The device determinesif the hands are being scrubbed 642. In one embodiment, the devicemonitors the environment near the sink to detect the scrubbing handsthrough sound and/or motion detection. If yes, the device determines ifall recommended scrubbing steps are completed in the recommended period647. In one embodiment, the period is twenty (20) seconds for thescrubbing steps. In another embodiment, the period is fifteen (15)seconds. Optionally, the process includes the output of a musicalcomposition, tune or song during the time period. The recommended periodis previously set during the configuration stage. If not, the devicecontinues to display recommended scrubbing steps 635. If yes, the rinseof hands step is displayed 650 on the device's display. Next, the devicemonitors the environment for the rinsing step 652. The device determinesif rinsing is detected 655. If this step is not detected, areinforcement message of rinsing is displayed 657.

Next, the step of the drying of hands is displayed 660 on the device'sdisplay. The device monitors the environment for the drying of hands662. The device determines if the step of drying of the hands occurs atstep 665. In one embodiment, the device listens for the use of an airdryer or paper towel dispenser. If no hand drying is detected, thedevice announces to the user to dry his/her hands 667. Once the devicedetects drying, the algorithm continues to then display instructions toshut the water off 670.

Finally, the user is congratulated with a successful completion messagedisplayed 672. Once the user is detected to have moved away from thesink, the device returns to a sleep state whereby the device displayconserves battery power by switching off or powering down the display.This detection of the user motion is determined by the sound of water nolonger pouring from the faucet; the closing of the faucet to the offposition; the user's footsteps moving away from the sink area; theclosing of the washroom; or some combination thereof. The process thenreturns to start 605 by monitoring the arrival of the next user.

FIGS. 7a-7m

Exemplary instructions for compliance with the recommended handwashingprotocol are shown in FIGS. 7a-7m . These instructions are sampleoutputs on display 320 on device 300. Similar outputs may be provided indevice 200. As shown in FIGS. 7c-7i , while the instructions areprovided on how to scrub the hands correctly, a timer, preferably in theform of a countdown, is outputted to help inform and createaccountability user to comply with the recommended scrubbing timeduration. The graphical countdown includes a graphical element 755 thateither increases or decreases according to the time remaining element760 in the countdown as illustrated in FIG. 7m . The steps illustratedare preferably embodied in an animated or video format with a voice-overand/or music.

FIGS. 8, 9 a-c

With reference to FIG. 8, the installation and set-up of device 300 nowfollow. The device 300 is preferably connected in the general vicinityof the sink area. In one variation, the display 320 is situated suchthat it is at a user's viewing angle when the user is washing his/herhands. One example of the installation is shown in FIG. 1c . Severalother locations of the device may accomplish the same objective,including placing the device on the counter of the wash sink or on anopposing wall facing the sink area.

It is within the scope of the present disclosure to include a pluralityof devices 300 working together collecting data that forms the learningtraining set for the plurality of devices. This technique is referred toas federated learning. This is a machine learning technique to train analgorithm across multiple decentralized edge devices (such as device300) or servers holding local data samples, without exchanging them.With the decentralization of servers, the entity managing the devicesand system of the present disclosure can also achieve data minimizationby reducing the amount of data that must be retained on a centralizedserver or in cloud storage. In that variation, a first device may housea set of the components described in device 300, while a second devicemay have another set of components described in device 300. For example,the second device may be positioned next to the sink area, primarilyacting as an output device with a display driven by the first devicepositioned in another area of the room through wireless communicationbetween the first and second device.

The configuration process includes the installer inputting the number ofwashroom elements at step 810. As shown in FIGS. 9a-9c , this input maybe achieved through a companion mobile app 900 in communication with thedevice 300. Elements may generally include any fixture or apparatus thatoriginates sound when in operation. Elements 905 may comprise sink,bathtub, shower, fan, doors, toilet, doors, the sound of footsteps onfloor and other elements. The installer then inputs the number of eachelement at step 820 found in the ambient environment and may add certaincharacterizations of each element to better distinguish a particularwashroom from other washroom environments. For example, the type of handair dryer or toilet could be inputted. At least one sample audiorecording for each element in operation 910 is preferably added at step830, and those labeled samples are added to the ML dataset at step 840to enhance the ML model detection of the user's hand hygiene compliance.The app requests and receives at least one or more of the sounds set outin FIG. 4 a.

In the same manner, labeled motions may be added to the ML dataset. Inone variation of the mobile app, the configurations of various physicallayouts of the environment where the device is installed is preloadedinto the app. This is likely feasible in commercial settings such asfast-food franchises where washroom configurations tend to bestandardized. In this manner, the companion app 900 may assist in thequick configuration of the device 300. The installer can quickly inputambient environment information through a simplified user interface (UI)as shown in FIGS. 9a -9 c.

Although actual data sampling of the ambient environment is describedabove, the present disclosure may utilize synthetic data. As the namesuggests, this type of data is artificially created rather than beinggenerated by actual events. It is often created with the help ofalgorithms and is used for a wide range of activities, including as testdata for new products and tools, for model validation, and in AI/MLmodel training. Synthetic data is important because it can be generatedto meet specific needs or conditions that are not available in existing(real) data. This can be useful in numerous cases such as:

-   -   when privacy requirements limit data availability or how it can        be used    -   data is needed for testing a product to be released however such        data either does not exist or is not available to the testers

In the present disclosure, in one embodiment, synthetic data enables theentity that manages the device 300 and related systems to use ambientrecord data while still maintaining user confidentiality. The syntheticdata may be combined with other PETs such as data masking andanonymization. The present disclosure includes modifying observed sensordata for purposes of adapting the training dataset towards a deploymentenvironment. In this way, even if the actual captured sensor data is notfrom that specific ambient environment, the data may be leveraged forthe target ambient environment. For instance, sensor data taken from arest room in a bus station may be adapted for use in a football stadium.When the model training takes place is also a consideration. In onevariation, the model may be trained prior to deployment, in anothervariation it may be trained over time in the deployment environment, orin yet another variation the model may be trained in a hybrid approachwhereby you start with a training data set that mimics the deploymentenvironment and then retrain/update the model after deployment, to adaptit to the specific target environment.

However, it must be noted that audio sounds may vary fromrestroom-to-restroom such as faucets (pressure of water), environmentalnoise (football stadium bathroom much louder compared to company officebathroom), etc. That being said, training specific models to adapt tocertain environments such as dedicating one model to loud environmentaldistortion noise (football stadium, bars, airport) is desired.

FIGS. 10a-10e

The process description below sets forth one analytic approach withoutthe use of ML described earlier to determine the type of element (toiletflush, fan, etc.) that is detected in the audio signal received by thedevice 300. It is within the scope of the present disclosure toincorporate some of the process below with the ML process descriptionabove. FIGS. 10a-10c are sample audio signal spectrum observations 1000associated with flushes from three different toilets. Samples werecollected over approximately a thirty second timespan. The vertical axisunits are decibels whilst the horizontal axis is the frequency of theaudio signal measured in Hertz for an element being monitored for itsaudio signal. The initialization/learning phase begins with measuringthe power of the element. The power, which is the energy in apredetermined frequency band, is measured. A power ratio, for theelement, is then established by dividing the power from the first bandfilter by the power from the second band filter. A ratio range isestablished for the element and variations thereof. A similar processoccurs for a second element. Once the learning phase is established forvarious elements, when the trained device 300 receives an audio signal,the power ratio of the signal is measured and the type of elementoriginating the audio signal may be determined by the association of theratio range established for an element.

By way of example, a first bandpass filter is set for 0-600 Hz, and asecond bandpass filter is set for 3000-5000 Hz. The power of the firstbandpass is divided by the power of the second bandpass to establish apower ratio. With reference to FIGS. 10a-10c , the power ratio of thetoilet flushes is approximately 3 through 7 (meaning the power in thefirst bandpass is 3 to 7 times more powerful than the second bandpass).Similarly, FIGS. 10d and 10e are sample audio signal observations 1000associated, respectively, with an exemplary bathroom fan and faucetrunning water into a sink. In contrast to the toilet flush, withreference to FIG. 10d , the fan tends to have a power ratio of greaterthan 7. Hence, if the device 300 determines a ratio of greater than 7,the origination of the audio signal is presumed to be a fan. Likewise,with reference to FIG. 10e , the faucet running water into the sinktends to have a power ratio less than 3. The above-described bands werechosen specifically for a specific device configuration (e.g., processorlimitations of the testing device); however, various bands may beadopted depending in part with the processor capabilities.

RFID Enabled Device

In one embodiment, the device has a Radio Frequency Identification(RFID) reader means. RFID uses radio waves produced by a reader todetect the presence of (then read the data stored on) an RFID tag. RFIDtags may be in small items like cards, buttons, or name tags onuniforms. This particular embodiment is useful for use in thehospitality or healthcare industry where employees are wearingRFID-enabled name tags or employee badges, and where collection of PIIas it relates to employees in their role at their work environment isacceptable. Employers are then able to enhance the compliance of theproper handwashing protocol of their employees when the device is inuse. If the device detects non-compliance, a notification is sent to theappropriate person or system to inform the particulars of the user'snon-compliance.

Gamification

In one variation, information is collected in the aggregate and storedon a datastore for dissemination in a way that illustrates the adherenceof certain washroom facilities compared to others. For example, ageographically diverse fast food business can readily identify whereproblematic users are located by the lack of adherence to therecommended steps. In an alternative variation, users may opt to sharetheir experience (on whatever social media platform of their choice) ofcompliance with proper handwash. They may do so by scanning a barcode orother unique identifier associated with the device to share stats of howwell that particular washroom is doing in having users comply withproperly washing their hands. In another variation, the device displaysa statistic to recognize the user's contribution to help fend off thespread of germs and disease by performing a proper handwash—for example:“Congratulations, you are 400th person today to properly handwash. Beclean, be safe and be healthy!”.

Voice Recognition and FIG. 4c

The present disclosure, in another variation, detects a user's voice asit only relates to issues associated with washroom hygiene-relatedissues. Some examples of the limited context of the voice it may detectand recognize are set forth in FIG. 4c . The intention of limiting therecognition of voice is to reduce the concerns relating to protectingthe user's privacy. The issues in FIG. 4c may trigger in this variationa new response by the device to modify the recommended handwashing stepsand/or seek to remedy the outstanding issue by informing the appropriatejanitorial staff.

Thermal Sensing Device

In another variation of the present disclosure, with reference to FIG.3a , an additional input device in the form of a thermal sensing devicemay be added to device 300. With adequate adaptation, this will addconsiderable new functionality in the form of better detecting userdeviation from the recommended handwashing protocol. The thermal sensingdevice will, for example, better distinguish when a user simply walkspast the device, perhaps to use the washroom facilities, including theurinal or toilet, versus leaving the washroom without attending to washhis/her hands. The thermal sensing device could combine audio data fromthe audio input device and/or motion-sensing device to further enhancethe positive determination of this deviation. Hence, this deviationdetection is directed to determine whether the user intends to use thesink to wash his/her hands. As an alternative to the thermal sensor, anoptical video capture device may be substituted. This may raise privacyissues that need to be addressed using PET.

In another aspect, a system to encourage hand hygiene comprises a firstdevice and a second device that cooperates with a toilet. The firstdevice is situated at or near the sink area while the second device issituated in or at the toilet tank area. The second device communicateswith the first device when the second device detects the flushingoperation of a toilet.

The second device comprising a microprocessor, power source, wirelesstransmitter, and sensor. The sensor may be an accelerometer, a motionsensor, or an infra-red sensor. The second device may comprise awaterproof or water-resistant housing that may be floatable on thesurface of water found in the toilet tank or cistern. Alternatively, thesecond device sits on the lip of the tank and detects the waterdisplacement. Detection could be sound, vibration, visual, infrared,motion, or some combination thereof.

In one aspect, the second device detects the displacement of water froma first level to a second level in the toilet tank or cistern after aflush operation occurs in the toilet. Upon this detection, the seconddevice wirelessly communicates information through the wirelesstransmitter to the first device about the flush operation. The firstdevice, upon receiving this information, activates its functionalitythat hand-hygiene related activity may soon ensue. Such functionalityincludes visual and/or audio annunciation to wash hands.

It should be noted that while certain steps within procedures may beoptional as described above, the steps are shown in FIGS. 5a, 5b, 6a, 6b, and 8 are merely examples for illustration, and certain other stepsmay be included or excluded as desired. Further, while a particularorder of the steps is shown, this ordering is merely illustrative, andany suitable arrangement of the steps may be utilized without departingfrom the scope of the embodiments herein. Moreover, while someprocedures are described separately, certain steps from each proceduremay be incorporated into each other procedure, and the procedures arenot meant to be mutually exclusive.

For example, while certain embodiments are described herein with respectto using certain models for purposes of non-compliance detection, themodels are not limited as such and may be used for other functions inother embodiments. In addition, while certain recommended handwashingprotocols are shown, other suitable protocols may be used accordingly.

The foregoing description has been directed to specific embodiments. Itwill be apparent, however, that other variations and modifications maybe made to the described embodiments, with the attainment of some or allof their advantages. For instance, it is expressly contemplated that thecomponents and/or elements described herein can be implemented assoftware being stored on a tangible (non-transitory) computer-readablemedium (e.g., disks/CDs/RAM/EEPROM/etc.) having program instructionsexecuting on a computer, hardware, firmware, or a combination thereof.Accordingly, this description to be taken only by way of example and notto otherwise limit the scope of the embodiments herein. Therefore, it isthe object of the appended claims to cover all such variations andmodifications as come within the true spirit and scope of theembodiments herein.

The invention claimed is:
 1. A method comprising: obtaining, by adevice, sensor data captured by one or more sensors deployed in awashroom, the sensor data comprising audio signals; identifying, by thedevice and using the sensor data as input to one or more machinelearning-based classifiers, one or more events associated with a user ofthe washroom; identifying, by the device and based on the sensor data, amaintenance issue associated with the washroom; modifying, by the deviceand based on the sensor data and on the identified maintenance issue,one or more steps of a hand hygiene protocol; after modifying the one ormore steps of the hand hygiene protocol, making, by the device, adecision as to whether the one or more events comply with the handhygiene protocol; and providing, by the device, data indicative of thedecision for presentation to the user.
 2. The method as in claim 1,wherein the sensor data further comprises motion signals indicative ofmovements by the user.
 3. The method as in claim 2, wherein the one ormore machine learning-based classifiers comprises a motion classifier.4. The method as in claim 1, wherein providing the data indicative ofthe decision for presentation to the user comprises: customizing thedata indicative of the decision based in part on a history of compliancewith the hand hygiene protocol by the user.
 5. The method as in claim 1,wherein the data indicative of the decision further indicates acomparison between the decision and other decisions made regardingcompliance with the hand hygiene protocol by others.
 6. The method as inclaim 1, further comprising: sending, by the device, an alert regardingthe identified maintenance issue.
 7. The method as in claim 1, whereinthe identified maintenance issue is selected from a group consisting of:no soap, no paper towels, a broken towel dispenser, a broken soapdispenser, a dirty washroom, and a dirty sink.
 8. The method as in claim1, wherein the sensor data comprises voice recognition data, and themaintenance issue is identified based on the voice recognition data. 9.The method as in claim 1, wherein the one or more sensors comprises aRadio Frequency Identification (RFID) reader.
 10. The method as in claim1, wherein identifying the one or more events associated with the usercomprises: obfuscating, by the device, personally identifiableinformation of the user that is present in the sensor data.
 11. Themethod as in claim 1, further comprising: providing, by the device, dataindicative of a step of the hand hygiene protocol for presentation tothe user, prior to identifying the one or more events.
 12. The method asin claim 1, wherein the audio signals are indicative of running water orof use of soap by the user.
 13. The method as in claim 1, wherein thedata indicative of the decision comprises an instruction to perform aparticular step of the hand hygiene protocol.
 14. The method as in claim1, further comprising: outputting, by the device, music via an audiooutput device for a period during which one or more steps of the handhygiene protocol are to be completed.
 15. The method as in claim 1,wherein the provided data comprises display data for presentation by anelectronic display, the display data comprising a time counting elementalongside a graphical indication of a step of the hand hygiene protocol.16. The method as in claim 1, wherein the one or more sensors deployedto the washroom comprises a thermal sensor.
 17. An apparatus comprising:one or more interfaces; a microprocessor coupled to the one or moreinterfaces; and a memory storing instructions that, when executed by themicroprocessor, are configured to: obtain sensor data captured by one ormore sensors deployed in a washroom, the sensor data comprising audiosignals; identify, using the sensor data as input to one or more machinelearning-based classifiers, one or more events associated with a user ofthe washroom; identify, based on the sensor data, a maintenance issueassociated with the washroom; modify, based on the sensor data and onthe identified maintenance issue, one or more steps of a hand hygieneprotocol; after modifying the one or more steps of the hand hygieneprotocol, make a decision as to whether the one or more events complywith the hand hygiene protocol; and provide data indicative of thedecision for presentation to the user.
 18. A tangible, non-transitory,computer-readable medium storing program instructions that cause adevice to execute a process comprising: obtaining, by the device, sensordata captured by one or more sensors deployed in a washroom, the sensordata comprising audio signals; identifying, by the device and using thesensor data as input to one or more machine learning-based classifiers,one or more events associated with a user of the washroom; identifying,by the device and based on the sensor data, a maintenance issueassociated with the washroom; modifying, by the device and based on thesensor data and on the identified maintenance issue, one or more stepsof a hand hygiene protocol; after modifying the one or more steps of thehand hygiene protocol, making, by the device, a decision as to whetherthe one or more events comply with the hand hygiene protocol; andproviding, by the device, data indicative of the decision forpresentation to the user.